Apparatus and method for multimode analytical sensing of items such as food

ABSTRACT

A multimode biological sample inspection apparatus and method is provided. The apparatus includes illumination hardware arrangement including transmission and sensing hardware, the illumination hardware arrangement configured to inspect a biological sample using at least two modes from a fluorescence imaging mode, a reflectance imaging mode, a scattering imaging mode, and a Raman imaging mode, and processing hardware configured to operate the illumination hardware arrangement according to a protocol including inspection settings of the at least two modes. The processing hardware receives scan results from the illumination hardware arrangement and identifies attributes of the biological sample. The processing hardware is configured to employ the attributes of at least one biological sample to alter the protocol.

The present application claims priority based on U.S. Provisional PatentApplication Ser. No. 62/645,514, filed Mar. 20, 2018, inventors DanielL. Farkas, et al., entitled “Devices and methods for multi-modeanalytical sensing, combinations of fluorescence, reflectance,scattering or Raman analysis of food samples,” the entirety of which isincorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates generally to the field of analyzing itemssuch as food, and more specifically, to the optical inspection of itemssuch as food for multiple issues, including but not limited tocomposition, quality and identity, spoilage, parasites, disease,adulteration, contamination, and other relevant issues and attributes.

Description of the Related Art

Food composition is of great interest. Assessing the composition offood, quantitatively and reproducibly, is beneficial in avoidingunintended and undesirable scenarios, ranging from a product not beingquite what it is stated to be (e.g., a lesser quality fish or olive oil)to intentional adulteration (for financial gain but also including byterrorist intent) to random contamination (such as by bacteria, some ofwhich may be lethal). These issues constitute the application domain of,respectively, food quality, food defense, and food safety. Given theplace food occupies in modern society, and the possible extremeimplications of any negative events, bringing the best testing to thetask of ensuring the quality and safety of the food supply is of greatinterest.

Unfortunately, some of the currently methods employed in food inspection(molecular/biochemical/biophysical, such as polymerase chain reaction(PCR), chromatography, mass spectrometry, etc.) are too slow to yieldusable results in real time, and due to the time required to perform ananalysis, tend to rely on random and very sparse sampling.

Optical imaging is an approach rapidly growing in popularity. Recentdevelopments have facilitated the production of smaller, less expensive,more efficient, and faster light sources and detectors. However, whenapplied to food samples, the accuracy of optical detection techniquescan be limited due to factors such as low penetration depth and lack ofcontrast, especially for low biomarker concentrations.

Recently more portable hand-held systems have been developed withlimited capabilities in, for example, food quality, safety, andadulteration applications. Machine Vision/RGB Color imaging is thesimplest form of measurement that examines visible range colorcharacteristics with low to moderate specificity and accuracy.Multicolor/hyperspectral imaging can offer compositional analysis beyondstandard machine vision. Typically, reflectance hyperspectral imagingcaptures more extensive and accurate data (and can generate more usefulinformation and spectral signatures) in the expanded wavelength rangefrom UV to Infrared, but still has relatively limited specificity. Thesesingle method systems are best suited for more mundane tasks such assorting color, size, and shape, or simple tasks such as identifyingforeign objects such as metal or plastic in food. Infrared imaging cancapture more unique chemometric data than the visible range (color),albeit at a higher cost.

It would therefore be beneficial to offer a device or method forinspecting items such as food, but also items such as plants, that avoidthe issues associated with previous designs. It would be particularlybeneficial to offer a device or method that can be used to quickly andefficiently assess and classify issues with items such as food,including issues such as composition, quality and identity, spoilage,parasites, disease, adulteration, contamination, as well as other issuesand attributes.

SUMMARY OF THE INVENTION

According to one embodiment, there is provided a biological sampleinspection apparatus, comprising an illumination hardware arrangementcomprising transmission and sensing hardware, the illumination hardwarearrangement configured to inspect a biological sample using at least twomodes from a group comprising a fluorescence imaging mode, a reflectanceimaging mode, a scattering imaging mode, and a Raman imaging mode, andprocessing hardware configured to operate the illumination hardwarearrangement according to a protocol comprising inspection settings ofthe at least two modes. The processing hardware receives scan resultsfrom the illumination hardware arrangement and identifies attributes ofthe biological sample. The processing hardware is configured to employthe attributes of at least one biological sample to alter the protocol.

According to a further embodiment of the present design, there isprovided a method for inspecting at least one biological sample,comprising determining a plurality of inspection modes for inspectingthe at least one biological sample using a multimode inspectionapparatus, determining an inspection protocol for inspecting the atleast one biological sample, wherein the inspection protocol comprisesinspection settings for the plurality of inspection modes, inspecting atleast one biological sample using the multimode inspection apparatusaccording to the protocol, and altering the protocol based on inspectionresults for multiple biological samples.

According to another embodiment of the present design, there is provideda biological sample inspection apparatus configured to inspect abiological sample for issues, comprising illumination hardwarecomprising transmission and sensing hardware configured to illuminateand sense attributes of the biological sample, the illumination hardwareconfigured to inspect the biological sample using multiple inspectionconfigurations from at least one of a fluorescence imaging mode, areflectance imaging mode, a scattering imaging mode, and a Raman imagingmode, and processing hardware configured to operate the illuminationhardware according to a protocol comprising inspection settings for themultiple inspection configurations, wherein the processing hardwarereceives scan results from the illumination hardware and identifiesattributes of the biological sample. The processing hardware isconfigured to employ the attributes of at least one biological sampleand alter the protocol based on the attributes of the one biologicalsample.

These and other advantages of the present invention will become apparentto those skilled in the art from the following detailed description ofthe invention and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present disclosure, referenceis now made to the following figures, wherein like reference numbersrefer to similar items throughout the figures:

FIG. 1 illustrates one embodiment of the present design includingillumination hardware for use with multiple modes of inspection;

FIG. 2 shows operation in spectral imaging analysis;

FIG. 3 shows the conceptual methods employable and employed inembodiments of the present design;

FIG. 4 illustrates a representation of a device used to inspect samplesin accordance with the present design;

FIG. 5 represents a potential embodiment of a handheld multimodeinspection system according to the present design;

FIG. 6 is a conceptual representation of artificial intelligenceemployable or employed in and/or with the current design;

FIG. 7 illustrates biological samples, specifically fish, andreflectance spectral signatures of the biological samples;

FIG. 8 illustrates biological samples (fish), and florescence spectralsignatures of the biological samples

FIG. 9 is an example embodiment of an internal artificial intelligencemodel;

FIG. 10 is a conceptual overview of operation according to the presentdesign;

and

FIG. 11 shows an alternate embodiment of the processing according to thepresent design.

DETAILED DESCRIPTION

The following description and the drawings illustrate specificembodiments sufficiently to enable those skilled in the art to practicethe system and method described. Other embodiments may incorporatestructural, logical, process and other changes. Examples merely typifypossible variations. Individual elements and functions are generallyoptional unless explicitly required, and the sequence of operations mayvary. Portions and features of some embodiments may be included in, orsubstituted for, those of others.

The present design comprises an analytical multimode optical system thatprovides at least two methods, including but not limited to fluorescenceimaging/spectroscopic, reflectance imaging/spectroscopic, scatteringimaging/spectroscopic, and/or Raman imaging/spectroscopic analysis of asample using at least one transmission source, such as a light source,and potentially a sensor or collection element. According to the presentdesign, there is provided an incident light beam, for example, travelingto a sample via a wide field, or point illumination at the same regionor periphery of the collection optics. The system then collects theemitted light from the sample and forwards the emitted light to aphotodetector and/or camera for analysis and viewing. The sample canalso be illuminated with a laser to generate Raman emissions,fluorescence emissions, or a scattering pattern (e.g. speckle), whichare then collected through the same or different optical path andprovided to a spectrograph or sensor for wavelength identification. Thepresent design employs multiple modes of inspection, and the mode ormodes employed depend on the specimen being inspected. Further, thepresent design employs feedback to cross-validate or verify differentmodes of operation and may in some instances be used to adjustinspection parameters.

The design in one instance includes an inspection tool and an associatedmethod of inspection. Specifically, the design may take the form of ahand-held or possibly small tabletop design, that incorporates multipletransmission and/or sensing devices, such as light emitting and sensingdevices, multiple corresponding spectral detection systems, andcommunication and analysis devices and methods. Such a tool enhances theability of an on-site inspector to analyze and communicate thecompositional, molecular, and chemical constituents of a targetedobject/specimen or in a targeted area.

In another aspect, the present design includes an inspection tool and anassociated method of inspection. Specifically, the design may include aspecialized tabletop, desktop, or hand-held tool that incorporatesmultiple lighting devices, multiple corresponding spectral detectionsystems, and a communication and analysis methods. The tool enhances theability of an on-site inspector to analyze and communicate thecompositional, molecular, and chemical constituents of a targeted objector in a targeted area.

As noted, optical imaging is rapidly growing in popularity due totechnological advances that have enabled the production of smaller, lessexpensive, more efficient, and faster light sources and detectors. Thesenew technologies have facilitated the acquisition of more accurateoptical image sets, yielding molecular, structural and physiologicalinformation from targeted samples. There are many different opticalmeasurement techniques used by industry and academic researchers alike,with each technology usually focusing on a specific property of light(intensity, polarization, wavelength, coherence, temporal change, etc.).However, no single method can rapidly and efficiently provide acomprehensive analysis of items such as food.

As used herein, the term “food” may be employed, and many of thediscussions and examples provided relate to fish. However, the design isnot so limited, and may be employed with virtually any biological sampleor item, including but not limited to animals and plants, with animalsincluding all types of animals, including but not limited to mammals,birds, fish, amphibians, and reptiles, and plants including any type ofplant, including but not limited to vegetables, fruits, legumes,flowers, tubers, roots, trees, as well as seeds, nuts, and the like. Thedesign is not limited to the various samples listed herein, nor limitedto food per se, but may be employed advantageously with virtually anytype of biological material may be inspected using the currentteachings.

When inspecting food samples, prior designs have been limited due tofactors such as low penetration depth and lack of contrast, especiallyfor low biomarker concentrations. The present design employs a strategiccombination of multiple optical detection technologies in an opticalsystem that becomes multimode, improving chemical and/or biologicaldetection accuracy. Each individual detection method can provide aspecific and complementary (sometimes synergistic) piece of informationregarding the sample being examined Thus, by combining a number of thesemethods, the impact of the individual limitations is minimized, and thecombined strengths of the multiple mode results can deliver highlyspecific results.

The advantages of multimode optical imaging include greatly reducing thetime required for the initial detection and enumeration of contaminants,with minimal sample preparation, nondestructive evaluation, fastacquisition times, and visualization of the spatial distribution ofnumerous components simultaneously. These advantages are highly usefulin detecting contaminants in food and assessing safety and quality.

The multiple modes may include but are not limited to the various modespresented and discussed below.

Hyperspectral Imaging (HSI) functions by integrating conventionalimaging and spectroscopy to gain spatial and spectral information froman object. HIS is capable of capturing reflectance, transmittance, andfluorescence images in the visible and infrared regions withsub-millimeter spatial resolution and high spectral resolution (10 nm).While HSI was originally developed for remote sensing, it has gainedpopularity in the field of food safety and analysis with newapplications reported in fruits and vegetables, poultry, and meat.Advantages HSI provides in comparison to other techniques (such as RGBimaging, NIR spectroscopy, and multicolor imaging) include the abilityto produce spatial and spectral information, multi-constituentinformation, and sensitivity to minor components.

HSI in the near infrared (NIR) can provide chemical composition aboutmeat, such as prediction of fat, protein, and water content of lambmeat. Moreover, HSI enables the detection of certain bacteria in food,such as E. coli.

Fungal growth on food products is of particular concern due to thepotential for detrimental effects on population health ranging fromallergic reactions and respiratory problems to the production ofmycotoxins. HSI has been deployed to identify fungal species such asAspergillus flavus, Aspergillus parasiticus, Aspergillus niger andFusarium spp. which can produce mycotoxins, which are secondarymetabolites that are toxic for humans and animals.

A common source of contamination for fresh produce and other rawmaterials used to produce food is fecal contamination. Multispectraldetection of fecal contamination on apples using HSI imaging has beendemonstrated. A HSI system with a range of 450 to 851 nm has been usedto examine reflectance images of experimentally contaminated apples.Fecal contamination sites may be evaluated using principal componentanalysis (PCA) with the goal of identifying two to four wavelengths thatcan be used in an online multispectral imaging system. Actual testinghas shown that contamination can be identified using either of threewavelengths in the green, red, and NIR regions.

With the use of HSI in the spectral range of 400-1000 nm, E. coli loadsin grass carp fish have been measured to evaluate microbial spoilage.Researchers have demonstrated that reflectance HSI in combination withmultivariate analysis had the ability to rapidly and non-invasivelyquantify and visualize the E. coli loads in grass carp fish flesh duringthe spoilage process. Distribution maps of samples examined using HISmay be found in Cheng, J. H., and Sun, D. W., “Rapid quantificationanalysis and visualization of Escherichia coli loads in grass carp fishflesh by hyperspectral imaging method,” Food and Bioprocess Technology,8(5), 951-959 (2015), the entirety of which is incorporated herein byreference, illustrating E. coli contamination. Such distribution mapsprovided detailed information of postmortem spoilage development ingrass carp flesh.

One of the main advantages that HSI has over conventional spectroscopymethods is its ability to provide visual distribution maps ofcontamination in a pixel-wise manner Multiplication of regressioncoefficients of a multiple linear regression model by the spectrum ofeach pixel in the image provides a prediction map showing thedistribution of E. coli within the fish flesh. In the Cheng and Sunreference, different E. coli loads are graphically represented by colorsfrom blue, (representing low or no bacteria growth) to red (representinghigh bacteria growth).

HSI is a non-destructive tool for direct, quantitative determination ofEnterobacteriaceae loads on chicken fillets. Such a process employspartial least squares regression (PLSR) models and root mean squarederrors. Such use of HSI entails a simplified PLSR model that predictsEnterobacteriaceae loads in every pixel of the image acquired from HSI,resulting in a new image called a ‘prediction map.’ The prediction mapuses a color scale to represent describe the different microbial loadsin each spot of the sample.

Feng, Y. Z., ElMasry, G., Sun, D. W., Scannell, A. G., Walsh, D., andMorcy, N., “Near-infrared hyperspectral imaging and partial leastsquares regression for rapid and reagentless determination ofEnterobacteriaceae on chicken fillets,” Food Chemistry, 138(2),1829-1836 (2013) shows an image of a median-filled prediction map usinga simplified PLSR model built on wavelengths of 930, 1121, and 1345 nm.The entirety of the Feng et al. reference is incorporated herein byreference. The values under each sample represent predictedEnterobacteriaceae counts in log₁₀ CFU g⁻¹. As shown in the Feng et al.representations, when the microbial loads increase, the images shiftfrom a blue color to a more reddish one, reflecting the growth ofbacteria on the chicken fillets.

Changes in temperature during cold storage of meat products can alsolead to undesirable microbial growths, which may affect food safety. HSIhas been used to measure biochemical changes within fresh beef. HSI hasshown potential for real-time and non-destructive detection of bacterialspoilage in beef.

HSI in the near-infrared range (900-1700 nm) has been used to determinethe total viable count and psychotropic plate count in chilled porkduring storage. In one instance, researchers captured NIR hyperspectralimages in the reflectance mode every 48 hours from each sample. Theresearchers assumed that meat spoilage would be evident at a microbialload of 107 CFU per gram or cm² and established a cut-off point of 106CFU/g as an acceptable threshold of freshness. Differences were observedin the wavelength range between 1300 and 1600 nm, where fresh sampleshad lower absorbance than spoiled samples, where results are shown inBarbin, D. F., ElMasry, G., Sun, D. W., Allen, P., & Morsy, N.,“Non-destructive assessment of microbial contamination in porcine meatusing NIR hyperspectral imaging,” Innovative Food Science & EmergingTechnologies, 17, 180-191 (2013), the entirety of which is incorporatedherein by reference. This spectral region (wavelength range between 1300and 1600 nm, is commonly assigned to N—H stretch of proteins (amines andamides) and their interactions with water. This spectral region suggeststhe occurrence of proteolytic changes, the main indicator for the onsetof spoilage in meat products.

Another use of HSI is fluorescence HSI coupled with multivariate imageanalysis techniques utilized for the detection of fecal contaminates onspinach leaves. Violet fluorescence excitation may be provided at, forexample, 405 nm with light emission was recorded from 464 to 800 nm.Partial least square discriminant analysis (PLSDA) and wavelength ratiomethods may be compared for detection accuracy for fecal contamination.In one study, the PLSDA model had 19% false positives for non-fresh poststorage leaves. A wavelength ratio technique using four wavebands (680,688, 703 and 723 nm) has been used to identify 100% of fecalcontaminates on both fresh and non-fresh leaves.

Detection of fecal contamination on cantaloupes using HS fluorescenceimagery has been employed. HS images of cantaloupes artificiallycontaminated with a range of diluted bovine feces have in one instancebeen acquired from 425 to 774 nm in response to ultraviolet-A (320 to400 nm) excitation. Evaluation of images at emission peak wavelengthsindicate that 675 nm exhibits a greatest contrast level betweencontaminated and untreated surface areas. Two-band ratios compared withthe single-band images enhanced the contrast between the fecalcontaminated spots and untreated cantaloupe surfaces.

Methods may also be employed to classify fecal contamination on leafygreens. Such methods may, for example, utilize HS fluorescence imagingsystem with ultraviolet-A excitation (320-400 nm) to provide detectionof bovine fecal contaminants on the abaxial and adaxial surfaces ofromaine lettuce and baby spinach leaves. For both lettuce and spinach,the detection of fecal matter may best be obtained using the ratio ofthe signal from 666 nm divided by that from 680 nm, with values of 0.98for romaine lettuce and 0.96 for baby spinach representing a levelindicating contamination.

Another technique that may be employed in the current design is RamanSpectroscopy and Spectral Imaging. Raman spectroscopy is anon-destructive spectroscopic technique, based on the vibrationalproperties of the constituent molecules, that provides molecularinformation about the sample under examination. The Raman signal resultsfrom molecules excited by a small amount of incident light at a specificwavelength. The remitted light has some photons shifted to differentwavelengths by the addition or subtraction of vibrational energy fromsome of the tissue intra-molecular bonds. Contrast is achieved when thetissue molecular constituents differ such that the Raman signals fromtwo tissues have different wavelength distributions. Raman SpectralImaging (RSI) intertwines Raman spectroscopy and digital imaging tovisualize the composition and structure of a target, which is useful infood safety and analysis. Historically, Raman imaging systems have onlybeen able to perform Raman measurement at a microscopic level. Suchsystems were unable to evaluate whole surfaces of individual foods.Recent studies have shown a benchtop point-scanning Raman chemicalimaging system designed and developed for food safety research. Althoughits signal-to-noise is low, Raman imaging is a highly specific andsensitive technique that allows for the detection of particularchemicals at low concentrations, such as melamine particles in dry milk.

One study aimed at the detection and differentiation of important foodand waterborne bacteria (E. coli, Staphylococcus epidermidis, Listeriamonocytogenes, and Enterococcus faecalis) used surface-enhanced Ramanspectroscopy (SERS) coupled with intracellular nanosilver as SERSsubstrates. Variations observed in the spectral patterns of bacterialpathogens resulted from different quantity and distribution of cellularcomponents such as proteins, phospholipids, nucleic acids, andcarbohydrates. SERS coupled with statistical analysis is useful indiscriminating and detecting bacterial cells, spores, and viruses.

A portable Raman sensor system has been presented with an integrated 671nm microsystem diode laser as excitation light source for the rapid insitu detection of meat spoilage and bacteria. The system used in thissituation demonstrates a reduction in form factor enabled by recentadvances, where such a system includes three main components: a handheldmeasurement head with a laser driver electronics board, the Ramanoptical bench, and a battery pack. Such a system has been employed torapidly detect meat spoilage in specific pork cuts, musculus longissimusdorsi (LD) and musculus semimembranosus (SM). The total number ofmesophilic aerobic microorganisms on the surface of the meat exhibitpossible correlations of bacterial growth with the measured Ramanspectra. Concentrations of melamine have been successfully measured inthis manner in wheat gluten, chicken feed, and processed foods such ascake and noodles.

Another technique potentially employed in the present design is Speckle(scattering) Imaging. Undesirable microorganisms capable of causingspoilage and poisoning include bacteria, yeast, and mold. Laser speckleimaging has been introduced in this field of application to monitormoving particles in optically inhomogeneous media by analyzingtime-varying laser speckle patterns in assessing meat quality anddetecting contaminants. Unlike multiple light scattering in meat, whichexhibits static and deterministic speckle intensity patterns, lightpaths associated with the movements of living microorganisms result intime-varying changes in speckle intensity patterns. By detecting thedecorrelation in the laser speckle intensity patterns from tissues, theliving activities of microorganisms can be detected.

Another advantage of speckle imaging is the ability to examine meatssealed with transparent packaging, as this method detects time-varyingsignals in reflected laser beams without loss of fidelity due totransparent plastic. Bacterial colonies can be detected within a fewseconds using speckle imaging. The method provides an efficient andeffective way to detect live bacteria in food products. Speckle imagingsystems sense the presence of bacterial colonies and other contaminantsin both food and water

One study detected and quantified various levels of contamination usingchicken breast meat samples. Meats contaminated with bacteria hadsignificant decreases in autocorrelation values over time, whereas thecontrol group (meat treated with a PBS solution) did not show any majorchanges. The meat treated with a high concentration of bacteria had moresignificant changes over time compared with the meat treated with a lowconcentration of bacteria. Moreover, the decrease in the autocorrelationvalue was proportional to the concentration of the treated bacteria. Themeasured autocorrelation values were all statistically different fromone another (p<0.001) and decreases in autocorrelation were proportionalto the concentration of bacteria. Thus spontaneous bacterial activitycaused strong decorrelation in laser speckle dynamics.

Yoon J, Lee K, Park Y., “A simple and rapid method for detecting livingmicroorganisms in food using laser speckle decorrelation,” arXivpreprint arXiv:1603.07343, 2016 Mar. 18. illustrates assessing bacterialactivity in meat. The Yoon et al. reference, the entirety of which isincorporated herein by reference, shows representative autocorrelationamps in meat treated with various concentrations of bacteria at varioustime lags, including averaged C(tau) values as a function of the timelag, as well as quantification of the autocorrelation values at tauequal to 10 seconds.

One label-free bacterial colony phenotyping technology is the BARDOT(Bacterial Rapid Detection using Optical Scattering Technology) system,which can provide classification for several different types ofbacteria. A certain speckle formation allows for the detection andidentification of these bacterial species. As the center diameter of theBacillus spp. colony grows from 500 to 900 microns, the average specklearea decreases two-fold in certain experiments and the number of smallspeckles increases seven-fold. As Bacillus colonies grow, the averagespeckle size in the scatter pattern decreases and the number of smallerspeckle increases due to the swarming growth characteristics of bacteriawithin the colony. Real-time detection and identification of Salmonellacolonies grown from inoculated peanut butter, chicken breast, andspinach or from naturally contaminated meat using BARDOT technology (90to 100% accuracy) in the presence of background microbiota fromnaturally contaminated meat.

The present design may be applied in various areas, including but notlimited to the following. In the field of seafood safety, Salmonellaenterica and Escherichia coli are members of the Enterobacteriaceaefamily and are widely found in the environment. They typically spread tohumans through the fecal-oral route or contact with contaminated watersources. While most strains of E. coli are harmless, there are severalpathogenic variants that cause gastrointestinal illness and can lead tohealth complications. Pathogenic E. coli are associated with foods suchas animal products and fresh produce. Because E. coli are abundant inhuman and animal feces, tests for generic E. coli levels in food areoften used to indicate recent fecal contamination or unsanitaryprocessing. These tests have been performed using culture-based methodsthat require 24-48 hours using rapid techniques or 5-7 days withtraditional means.

S. enterica is the leading bacterial cause of foodborne illness in theUnited States and is associated with a variety of foods, includingpoultry, fresh produce, dairy products, and some low-moisture foods.Detection of S. enterica in foods using traditional culture methods islaborious and time-consuming, requiring 4-5 days for confirmation of apositive sample. Rapid detection methods, such as real-time polymerasechain reaction (PCR), have been developed for S. enterica. These methodscan typically be completed within about 1-3 hours. However, theygenerally require at least a primary enrichment step and sometimes aselective enrichment step, which can add up to 24-48 hours to theprocedure.

The use of multimode hyperspectral imaging (HSI) as a rapid screeningtechnique reduces the time required for the initial detection andenumeration of contamination on foods. Hyperspectral imaging integratesspectroscopic and imaging techniques to enable direct identification ofdifferent components and their spatial distribution in the testedsample. The resulting three-dimensional dataset or ‘hypercube’ containstwo spatial dimensions and one spectral dimension. The advantages ofhyperspectral imaging over traditional methods include no/minimal samplepreparation, nondestructive nature, fast acquisition times, andvisualization of the spatial distribution of numerous componentssimultaneously. Previous studies have utilized hyperspectral imaging todetermine quality and microbial characteristics of food products andcontact surfaces.

HSI combined with chemometrics is employable to detect and enumerate E.coli on fresh spinach. Near-infrared HSI can be used to enumerateEnterobacteriaceae on chicken fillets and in certain instances candetect bacterial loads at levels of 2.4-5.2 log CFU/g. Hyperspectralfluorescence imaging has also been used to detect biofilms of S.enterica and E. coli O157:H7 on food contact surfaces.

Fluorescence spectroscopy has been used previously to identify anddifferentiate foodborne bacteria. Fluorescence spectroscopy is a simple,non-destructive, non-invasive and relatively inexpensive analyticalmethod. In comparison with other classical analytical methods,fluorescence spectroscopy provides enhanced selectivity, highsensitivity to a wide array of potential analytes, and has norequirement for consumable reagents or extensive sample pre-treatment.This technique is based on the intrinsic fluorescence of bacterial cellcomponents. When examined with ultraviolet light, aromatic amino acidresidues (tryptophan, tyrosine, phenylalanine), nucleic acids, andco-enzymes are intrinsic fluorophores. However, due to themulticomponent nature of items such as foods, their fluorescence spectraare complex and chemometric methods using multivariate analysis areneeded to extract contaminant specific information. The present designmay vary both the excitation and detection wavelengths, and measure bothreflectance and fluorescence emission properties of a food sample. Thesystem is adjusted or may adjust for specific foods and contaminants.For biological tissues, dual or multiple excitation fluorescence canincrease the specificity and accuracy of classification andquantification of specific sources of fluorescence. When the systememploys dual excitation wavelengths, fluorescence emission contributionsof food contaminants can be more precisely detected, and the system maydisregard other irrelevant or unnecessary fluorescence components of thefood sample. Ratiometric versions of this approach may be employed.

Laser Speckle Contrast Imaging (LSCI) is a wide field of view,non-scanning optical technique used in observing, for example, bloodflow in medical applications or live bacteria colonies in food samples.Speckles are produced when coherent light scattered back from biologicaltissue is diffracted through the limiting aperture of focusing optics.Mobile scatterers, i.e. scattering objects or items that are moving,cause the speckle pattern to blur. The present design may employ a modelthat inversely relates the degree of blur, called “speckle contrast,” tothe scatterer speed. In tissue, red blood cells are the main source ofmoving scatterers. Bacteria movement acts as a virtual contrast agent.

In the case of seafood adulteration, for example, the present system isa multimode hyperspectral image acquisition system for real-timeassessment of fish quality and adulteration. With increased seafoodimports and limited monitoring, fraud and deception in seafood marketingis a growing food safety concern. A desired high quality fish may besubstituted with a lower quality, less costly fish unbeknownst to thepurchaser and/or consumer. The flesh of many fish species is similar intaste and texture and, therefore, identifying species in fillet form canbe challenging, especially after preparation for consumption. One surveyby the National Marine Fisheries Service's National Seafood InspectionLaboratory (NSIL) found that over a nine-year period, 37% of fish and13% of other seafood (e.g., shellfish, edible seaweed) from randomlyselected vendors were mislabeled.

The present design overcomes the limitations of previous spectroscopicsolutions focused on fish quality and authentication through a number ofimprovements, the main one being the use of a multimode imaging approachthat combines multiple imaging methods. One combination is reflectanceand fluorescence hyperspectral imaging. Hyperspectral imaging detectsvarious types of fish with high accuracy (i.e. wild versus farmedsalmon). The present design identifies and validates key wavelengthbands that are central to fish quality assessment and authentication.

In the area of meat adulteration, many individuals worldwide follow adiet that restricts them to eating only halal food products. Dietaryrestrictions such as halal, have several guidelines to follow. IslamicLaw does not allow Muslims to eat or use any product derived from pig.Moreover, halal consumers have become concerned about issues such aspork substitution, undeclared blood plasma, use of prohibitedingredients, and non-halal methods of slaughter, among other concerns.One of the main authenticity issues which is common among Muslimconsumers is the need to determine whether meat products from halalspecies have been mixed with similar material from a cheaper non-halalspecies. Food manufacturers sometimes choose to substitute porkderivatives in food products as they tend to be cheaper and readilyavailable. Such pork derivatives may include pork tissues (e.g. collagenand offal), porcine mechanically recovered meats (MRM) and pork fat(lard). Animal fat from one species is often fraudulently used tosubstitute animal fat from another species. If the substitution containspork fat, then that product becomes ‘haram,’ or forbidden by Islamiclaw. Another form of substitution of meat products is the use ofmechanically recovered meat (MRM). MRM describes the residual materialoff bones that is obtained by machines operating on hydraulic or otherpressure principles in such a way that the structure of the material isbroken down enough for it to flow in a paste-like form from the bone.Chicken and pork carcasses are the most commonly used material for MRMproduction today. If pork carcasses are used, the resultant productwould be considered haram and not for consumption by Muslim consumers.

The food industry uses porcine blood and its derivatives, plasma and redcells, as food ingredients. Any product where blood is added isunacceptable for Muslim consumers. The need exists for techniques thatcan determine the authenticity of halal food products.

Several analytical methods that have been developed to detectadulteration rely on protein or DNA analysis. Some protein analysismethods include ELISA, chromatography, and FTIR spectroscopy. Analysisof DNA is typically done through polymerase chain reaction (PCR). ELISAbased techniques using polyclonal antibodies have several disadvantagesincluding limited production, variable affinity, and the requirement forextensive purification procedures to eliminate cross-reactivity for aparticular species identification. Methods such as PCR can typically becompleted within 1-3 hours. However, they generally require at least aprimary enrichment step and sometimes a selective enrichment step, whichcan add up to 24-48 hours to the procedure. Moreover, DNA extractionfrom food products has several problems and limitations. Food productsare made up of carbohydrates, fat, and chemicals that are ofteninhibitory to the PCR reaction, leading to false results (negative orpositive). Furthermore, the standardization of the PCR procedure(sampling to result interpretation) is difficult and requires specificskills and cautious handling to complete. Also, conventional PCR has thelimitation of not providing information on the quantitative analysis ofthe food product. Thus, many of these techniques require long waitingperiods to obtain results and therefore are not suitable for rapidassessment of food authenticity.

Multimode hyperspectral imaging (HSI) as a rapid screening techniquegreatly reduces the time required for detection of prohibitedingredients in halal foods. Hyperspectral imaging integratesspectroscopic and imaging techniques to enable direct identification ofdifferent components and their spatial distribution in the testedsample. The resulting three-dimensional dataset or ‘hypercube’ containstwo spatial dimensions and one spectral dimension. The advantages ofhyperspectral imaging over traditional methods include no, or minimal,sample preparation, no contact, nondestructive nature, fast acquisitiontimes, and visualization of the spatial distribution of numerouscomponents simultaneously.

The design presented, using multimode imaging, circumvents limitationsof other analytical methods. Again, the current design combines severaloptical imaging methods, for example a combination of reflectance andfluorescence spectroscopy together with dynamic speckle imaging.Fluorescence spectroscopy is a simple, non-destructive, non-invasive andrelatively inexpensive analytical method that provides enhancedselectivity, high sensitivity to a wide array of potential analytes, aswell as no requirement for consumable reagents or extensive samplepre-treatment. Fluorescence spectroscopy is based on the intrinsicfluorescence of bacterial cell components. When examined withultraviolet light, aromatic amino acid residues, nucleic acids, andco-enzymes are intrinsic fluorophores. However, due to themulticomponent nature of foods, their fluorescence spectra are complex,and chemometric methods using multivariate analysis are employed toextract contaminant specific information. By varying both the excitationand detection wavelengths and measuring both reflectance andfluorescence emission properties of a food sample, the system can beemployed to accurately assess specific foods and contaminants. Forbiological tissues, dual or multiple excitation fluorescence canincrease the specificity and accuracy of classification andquantification of specific sources of fluorescence. The present systememploying dual excitation wavelengths allows for more specific detectionof fluorescence emission contributions of food contaminants anddisregard other fluorescence components of the food sample.

Prior concepts, designs, and studies have utilized a single opticaltechnique to successfully determine authenticity, for example. Multimodeoptical imaging in accordance with the current design provides greateraccuracy in less time. Infrared spectroscopy, for example, is a fast,sensitive, and non-destructive technique that may be used to analyzefood products for authenticity studies. Analyzing a food sample usingthe mid infrared spectrum (4000-400 cm⁻¹) can give valuable informationabout the existence of molecular bonds. Such details can help determinethe types of molecules present in the food. Pig derivatives (such aslard) in any food product are prohibited by halal food consumers. Thecurrent system may employ Fourier Transform Infrared Spectroscopy (FTIR)combined with attenuated total reflectance (ATR) and partial leastsquare regression (PLSR) to detect the presence of lard in food items.

The FTIR spectra of both mutton body fat (MBF) and lard is shown inJaswir, I., Mirghani, M. E. S., Hassan, T. H., and Said, M. Z. M.,“Determination of lard in mixture of body fats of mutton and cow byFourier transform infrared spectroscopy,” Journal of oleo science,52(12), 633-638 (2003), the entirety of which is incorporated herein byreference. One representation in Mirghani, et al. shows distinctdifferences in the raw spectra obtained between MBF and lard. Thefrequency region 3010-3000 cm⁻¹ indicates a significant differencebetween lard and MBF. The lard spectrum has a sharp band at higherfrequency (3009 cm⁻¹) than MBF which has a shoulder peak at lowfrequency (3001 cm⁻¹). FTIR inspection provides a clear and concisemanner to identify lard in a mixture of other fats.

Another method employable in the current design is visible and nearinfrared reflectance spectroscopy (VIS-NIRS), which can be useful todiscriminate meat and meat juices from different livestock species. Inone trial, meat samples corresponding to beef, llamas, and horses werehomogenized and their spectra collected in reflectance (in the range of400-2500 nm). VIS-NIRS combined with partial least square regressionanalysis can be an accurate tool to discriminate meat obtained frombeef, llama and horse through analysis of the spectral data of mincedmeat, collected by reflectance.

NIR hyperspectral imaging technology may be employed to, for example, todetect adulteration in meat. Previous testing resulted in theidentification of four ‘important’ wavelengths later used to predict thelevel of adulteration in minced lamb meat. Spectral data collected fromNIR hyperspectral imaging combined with multivariate analysis can besuccessfully used to detect adulteration in meat.

Raman spectroscopy in combination with chemometrics can be employed forrapid determination of beef adulteration with horsemeat. Raman spectraof meat samples have been gathered at frequencies between 200 and 2000cm⁻¹. Spectral differences and unique bands that belong to horse fat maybe observed. These spectral differences between horse and beef are fromthe unique bands of horse fat at 919, 974, 1215 cm⁻¹. The system mayemploy these unique bands and may examine samples for bands that belongto horse fat in horsemeat and can develop a level of confidence that thesample of beef being examined has been adulterated with horsemeat. Someof the advantages of techniques such as Raman Spectroscopy over othernon-optical methods are short analysis time (in 30 seconds) and norequirement for time consuming sample preparation procedures. Boyaci, I.H., Temiz, H. T., trysal, R. S., Velioglu, H. M., Yadegari, R. J., andRishkan, M. M, “A novel method for discrimination of beef and horsemeatusing Raman spectroscopy,” Food chemistry, 148, 37-41 (2014) illustratesan original Raman spectra of horsemeat and beef samples, as well as thefirst derivative of the Raman spectra.

Several techniques have been proposed for detecting meat adulterationand food fraud, namely, for halal authentication. Each approach providesa rapid and accurate analysis of the sample being examined, enabling thedetection of any undesirable. The present approach integrates severaloptical imaging methods to bring about the highest level of accuracy andefficiency for food authentication. Thus, in present day, multimodeoptical imaging is the most advanced and well-equipped technique fordetermination of halal authenticity.

The present design therefore comprises a multimode hyperspectral imagingsystem. Due to the multicomponent nature of biological items such asfoods, their reflectance or fluorescence spectra are complex.Chemometric methods using multivariate analysis are employed by thepresent system to extract contaminant specific information. By varyingboth the excitation and detection wavelengths and measuring bothreflectance and fluorescence emission properties of a food sample,profiles may be assessed, refined, and employed when examining specificfoods and contaminants. For biological tissues, dual or multipleexcitation fluorescence can increase the specificity and accuracy ofclassification and quantification of specific sources of fluorescence.The combination of different spectroscopic methods (such as fluorescenceand NIR spectroscopy) circumvents single method inherent limitations andcan employ optical sensing for in situ mycotoxin detection. Additionalchemometric tools eliminate factors related to disturbing the specimenand enable extraction of desired biochemical information with respect tocontamination with fungi and/or mycotoxins.

The multimode hyperspectral imaging system may operate in fluorescenceand reflectance modes and may concurrently, or at a different time,employ speckle imaging. One example of such a system is presented inFIG. 7. The system uses spectral band sequential imaging on thedetection side. To ensure high signal to noise level, camera andspectral selection filter integration time is optimized for eachspectral band from visible to the near infrared. The illumination moduleuses two independent light sources to provide illumination forfluorescence excitation and reflectance measurements using threecomputer-controlled LED illumination rings. The UVA (375 nm) andblue/violet (420 nm) LED rings provide fluorescence excitation. WhiteLEDs will be used for reflectance illumination. The HSi-440C0Hyperspectral Imaging System (Gooch & Housego, UK, originally developedby ChromoDynamics, Inc.) incorporated in the proposed system can imageand analyze multiple signals in fixed and living cells at video rates.Its tunable filter can switch wavelengths within microseconds. Thesystem acquires multi-wavelength, high-spatial and spectral resolutionimage datasets, and can compute and display quantitative signal-specificimages in near real-time. The spectrally controllable image capturesystem can record spectral images of food samples in wavelengths rangingfrom 450 nm through 800 nm. The system is configured as a tabletopplatform where illumination and detection operate above the food sample.

In this system, time-varying speckle signals can be quantitativelyaddressed with speckle correlation time. A sample containing livingmicroorganisms has a correlation time shorter than a static one, andthus contaminated food is less time-correlated compared to fresh fooddue to the spontaneous motility of microorganisms. Correlation time ofscattered light from samples, as well as presence and activity ofmicroorganisms are quantitatively analyzed.

FIG. 1 includes a camera 101 at the top, acousto-optical tunable filter(AOTF) 102, lens 103, ring illumination element 104, laser (speckleimaging) arrangement 105, and sample holder 106 to hold the sample.Shown as representation 108 in the upper left corner are variousbiological sample scans representing scanning using different modes ofthe multimode apparatus. Spectral selection can be implemented inillumination optical path and/or detection optical paths. The methods ofspectral selection may include a filter wheel, an acousto-opticaltunable filter (AOTF), a liquid crystal tunable filter (LCTF), DMD basedspectral filter, Fabry Perot based spectral filter, and/or diffractionbased spectral filters such as gratings and prisms (not shown in thisview). Any or all of the imaging components can be offered in thismultimode imaging system.

Regarding processing the images received, consider I (x,y,t) the imageof the sample at time t. The correlation coefficient between two imagesof the sample at different times is given by the normalizedautocorrelation function:

$\begin{matrix}{{C\left( {x,y,t} \right)} = {\frac{1}{T - \tau}{\sum\limits_{t = 1}^{T - \tau}{{{I\left( {x,y,t} \right)} \cdot {I\left( {x,y,{t + \tau}} \right)}}\delta \; t}}}} & (1)\end{matrix}$

where T is the total acquisition time, δt the time difference, and τ thetime lag. In the case of food contamination assessment, the sample isconsidered and the correlation to be close to the unity. Everydecorrelation effect is therefore due to the presence of livemicroorganisms moving across the sample.

For image correlation and calibration, since each spectral band isrecorded at different exposure times to ensure optimized signal to noiseratios, the design employs a calibration method to correct forinstrument response (e.g. lens and illumination non uniformity) andexposure time variations. Flat field corrected reflectance image (RI)spectra will be calculated as follows:

$\begin{matrix}{{{RI}\left( {x,y,\lambda} \right)} = \frac{{{RS}\left( {x,y,\lambda} \right)} - {{RD}\left( {x,y,\lambda} \right)}}{{{RR}\; \left( {x,y,\lambda} \right)} - {{RD}\left( {x,y,\lambda} \right)}}} & (2)\end{matrix}$

where RS is the sample image, RD is the dark current image, and RR isthe reference image. The present design may employ, for example, WhiteSpectralon® reflectance and fluorescence targets (Labsphere Inc., NorthSutton, N.H.) to acquire the reference image. The same exposure timewill be used for each wavelength during RD and RR image acquisition.

For spectral data analysis the system employs an algorithm for datapreprocessing to extract fluorescence emission and reflectance spectraldata cube, an algorithm for multivariant analysis, and an algorithm forrelative quantification of food contaminant concentration in the sample.FIG. 2 shows operation involved in multimode hyperspectral imaginganalysis. The system may employ MATLAB software for image analysis. Thesystem selects wavelengths preserving the largest amount of energy amongthe multimode spectral data with the aid of multivariant analysis,providing maximum discrimination between samples with differentcontaminant concentrations. Image analysis is typically performedoff-line after image acquisition is completed. The system sortsessential wavelengths from whole spectral data using general visualinspection of spectral curves to more advanced objective approaches suchas correlation analysis, analysis of spectral differences from averagespectrum, stepwise regression, discriminant analysis, and principalcomponent analysis.

With respect to speckle image analysis, time-varying speckle signals canbe quantitatively addressed with the speckle correlation time. A samplecontaining living microorganisms has a correlation time significantlyshorter than a static one. As a result, contaminated food is less timecorrelated compared to fresh food due to the spontaneous motility ofmicroorganisms. The system quantitatively analyzes correlation time ofscattered light from samples and the presence and activity ofmicroorganisms.

If I(x,y,t) represents the image of the sample at time t, thecorrelation coefficient between two images of the sample at differenttimes is given by the following normalized autocorrelation function:

$\begin{matrix}{{C\left( {x,y,t} \right)} = {\frac{1}{T - \tau}{\sum\limits_{t = 1}^{T - \tau}{{{I\left( {x,y,t} \right)} \cdot {I\left( {x,y,{t + \tau}} \right)}}\delta \; t}}}} & (3)\end{matrix}$

where T is the total acquisition time, δt the time difference, and τ thetime lag. In the case of food contamination assessment, the sample isconsidered static and the correlation is expected to be close to unity.Every decorrelation effect is due to the presence of live microorganismsmoving across the sample, indicating an issue.

The system extracts diagnostic information from multimode/hyperspectraldatasets, such as by testing multiple (e.g. six) different spectralsegmentation algorithms on the same image data for optimaldiscrimination. Methods range from relatively simple, established ones(square Euclidean distances, principal components analysis) to morespecialized ones (Mahalonobis distances, support vector machines,multivariant analysis) and depend on circumstances.

From FIG. 2, the system employs a reference target at point 201, and maycalibrate using the reference target. At point 202, the system acquiresmultimode spectral images of the reference target and may calibrate inthis instance. At point 203, the system is provided with the sample, andat point 204, the system acquires multimode spectral images of thesample. The two images may again be calibrated at point 205, while atpoint 206 the system performs spectral data extraction andpreprocessing. Point 207 represents the spectral analysis performed,where point 208 represents quantitative analysis and point 209 ismultivariant analysis, including PCA, PLS, DA, PCR, and/or otheranalysis techniques. Point 210 employs dimensionality reduction andwavelength selection, provides selected data to image postprocessing anddata recognition at point 211, and determines final results at point212. Point 213 is certain further processing, including classification,identification, mapping, and/or visualization to convey results topersons. Point 214 represents laser wavelength selection for speckleimaging, where point 215 records a series of images. Point 216 providesspeckle analysis. Speckle imaging may be provided as an alternate to, orin addition to, the image postprocessing and other functions presentedat points 211, 212, and 213.

As may be appreciated, multiple modes may be employed in the currentdesign. While FIG. 2 shows spectral analysis and speckle analysis,different modes may be employed, and in certain instances, availablemodes may be turned off or not used. For example, if the specimen has aprofile wherein certain benefits may be obtained using speckle analysisand infrared imaging, other modes offered (speckle analysis, brightfieldimaging, and so forth) present in the device and available for use forinspection may not be employed, particularly if their use does not lendto improved results for the sample considered.

FIG. 3 represents using multimode optical imaging to characterize foodsamples. A series of optical technologies can be used based on thecomplexity of detection as well as acquisition speed, field of view,type of optical signature to be recorded. The system or a user mayselect one or more optical methods based on a specific detectionproblem. In some embodiments, at least two methods in each categoryshown in FIG. 3 may be used to enhance the accuracy of measurement. FromFIG. 3, the least accurate/specific techniques have the largest field ofview, have higher speed, and result in a lower cost, while the converseis true for those techniques presented at the bottom of FIG. 3. Indescending order, techniques include “Survey” imaging and sensing, suchas speckle imaging, UV imaging, etc., followed by “zoom in” imaging andsensing, including for example reflectance multiwavelength imaging,fluorescence imaging, and the like, followed by “diagnostic” imaging andsensing (molecular imaging and sensing using hyperspectral reflectanceand/or fluorescence), and “chemical or atomic” imaging and sensing, suchas Raman spectroscopy, imaging, FTIR methods, and point spectroscopy.

An alternate version of the design is presented in FIG. 4. From FIG. 4,specimen 401 is shown along with imaging optics 402. Different opticsand optical channels or paths may be provided. From there severaloptional imaging techniques are pictured in accordance with therepresentative imaging and sensing techniques of FIG. 3. It is to beunderstood that more techniques may be employed depending oncircumstances, and various imaging optics may be employed or in someinstances such as simple photography, no optical components besides thenamed apparatus may be required. Further, the presence of broken linesindicates that all of these components are optional, but in general,more than one technique is employed, with FIG. 1 representing onepossible multimode embodiment. From FIG. 4, the system may employ“survey” imaging and sensing components, such as smartphone imaging 403,color photography 404, speckle imaging 405, IR imaging 406, UV imaging407, and visual assessment 408. “Zoom in” imaging and sensing isrepresented by reflectance multiwavelength imaging 409 and fluorescenceimaging 410, while “diagnostic” imaging and sensing is represented bymolecular imaging/sensing using hyperspectral reflectance and/orfluorescence at point 411. “Chemical or atomic” imaging and sensing isrepresented by Raman spectroscopy/imaging 412, FTIR methods 413, andpoint spectroscopy 414. Such techniques may be provided concurrently orsequentially.

The present design may be provided in a tabletop or handheld form, andmay take different forms depending on sizing, as well as need for andavailability of components. FIG. 5 shows representations of a multimodeoptical imaging system as a portable platform. The handheld design showngenerates illumination for reflectance and or fluorescence using LEDring (polarized or unpolarized) techniques. Reflectance and/orfluorescence emission photons may be filtered by wavelengths orpolarization at the detection optical path.

Processing

The system characterizes materials, typically biological materials suchas food, drugs, etc. based on multimode spectral analysis. According tothe present design, such analysis includes identifying features fromdifferent modes of measurement. Feature extraction/selection strategy ormethods for different modes of measurement may differ based onmeasurement physics and biological/chemical characteristics. Examples offeature extraction methods include wavelet transform, startisticalfeatures, haralick textural features, fractal analysis, and curvelettransform. The system may employ feature selection methods such asprincipal component analysis (PCA), independent component analysis(ICA), curvature and/or manifold learning.

Wavelet transform is a mathematical transform to extract informationfrom a signal or an image. In one dimensional wavelet transform, theinput signal is represented at different scales called coarse and detailcomponents using a set of basis functions originally from a functioncalled mother function.

The system may, online or offline, include identifying which spectralmeasurement mode (or combination) will have the highest impact resultingamong the top combinations. Optimization is based on a cost function(sensitivity, specificity, area under the curve) from a receiveroperating characteristics (ROC) curve. Depending on technologycomplexity, the system employs desired practical modes.

Based on the practical modes determined, the present design links thebiology and chemical components of samples and correlates them withhighest differentiating spectral features. Different samples may exhibitdifferences in this regard, such as fish, beef, plants, fruits, and soforth. The system may run independent measurements using metabolicand/or chemical analysis of samples to validate the biological/chemicaldifferentiation between samples to determine optimal modalities.

The system uses these optimal modalities to conduct a pilot study withsample size greater than or equal to a number of samples, such as 100,sufficient for collection and analysis of enough data from a variety ofsamples in view of other parameters. In the case of fish, for example,the number of samples may include a number of fish species (for example,up to 15 species) and other parameters considered may include attributessuch as frozen and thawed versus fresh, farm-raised versus wild-caught,and so forth.

FIG. 6 illustrates the proposed architecture and high-level steps. Level1 includes Signal Conditioning, Feature extraction and Featureselection. Level 2 includes all trained models for differentapplications and classes. Level 3 fuses internal decisions (AI scores)coming from all internal AI models to get a final AI score. Level 4applies a receiver operating characteristic (ROC), representing thediagnostic ability of the Classifiers as the discrimination threshold isvaried, to enforce desired balance between specificity and sensitivityof the AI system.

Processing of this type may be divided into two modules, featureextraction and classification. The feature extraction module processesthe raw data into a low dimensional feature vector that is relativelyinvariant to distortions and artifacts and is high in informationcontent, making it suitable to be used by the Classifier stage. Priorknowledge about the data and the experience acquired on building similarsystems may be employed. This stage of machine learning (ML) may requireexperimentation and fine-tuning by hand. The Classifier is usuallychosen from the large number of available generic modules and is trainedusing available data.

The present system may employ a numerical computing framework, such asMATLAB, for model development and validation. The observations (inputdata) are raw measurements obtained by the system. Training set class“labels” may be provided by DNA analysis. In the fish example, data maybe collected from at least 15 types of fish and used for theclassification pipeline model selection and validation, and secondarilyused as a holdout set of some number, such as 100, fish for classifierfinal testing. In the first stage, internal Classifiers are trainedseparately. A final classifier is obtained by fusing the prediction ofseveral internal classifiers (models).

Thus the system employs spectral acquisitions (raw signals) where rawdata includes fluorescence and reflectance spectral data and correlativefish DNA results. FIG. 7 illustrates reflectance results while FIG. 8shows florescence results, representing typical raw data signals. Fishfillets are imaged on the left, with acquired spectral signatures shownion the right. Data may be collected from any number of cases and usedfor classification pipeline model selection and validation. A holdoutset may be employed for Classifier final testing.

The system performs signal conditioning to reduce data variabilitycaused by differences in hardware probes and operating conditions. Thisincludes probe-specific calibration, dark current removal, andwavelength alignment to a unique set of wavelengths via interpolation.In addition, the system uses signal-to-noise, signal validation andsaturation tests to reject bad data samples.

Data may be initially calibrated prior to feeding to the classificationpipeline, based on individual spectroscopic data acquisition systemcharacteristics. The extraneous parts of the signal may be truncatedfrom the reflectance and fluorescence signals. Valid wavelength rangesmay be obtained by examining raw data, automatically or manually, orperforming an optimization and exhaustive search to find validwavelength ranges.

The system may perform wavelength alignment using interpolation. Becausethe spectrometer cannot be calibrated such that the response is measuredexactly at the same wavelengths for all units, the system may employ areference wavelength grid to compare collected signals. The system mayobtain the signal aligned to the reference grid from the raw signalusing a cubic or linear spline interpolation from values measured by aspectrometer, and these values may be used by the pipeline.

After initial signal conditioning, the system may extract features fromthe raw data. Processing of conditioned data into a low dimensionalfeature vector creates features that are relatively invariant todistortions and artifacts and valuable informational content. Thiscombination makes the results of this stage suitable to be used in theClassifier stage.

The system, or those providing functionality for the system, may useprior knowledge about the data during this stage to determine optimalmethods of feature extraction. These include original raw data in linearor log space, wavelet transform, statistical features, and texturalfeatures. The system may use feature level fusion by combining into asingle vector feature vector. The system may determine or provide aseparate AI model for each feature type and will fuse outcomes of eachAI in decision levels as well.

The system may perform feature selection using methods such as Principalcomponents analysis (PCA) and Independent Component Analysis (ICA)algorithms for dimensionality reduction and removing redundant featuresand information.

PCA is a statistical method that converts a set of observations andsensor data with some level of redundancy and correlation into a set ofuncorrelated components called principal components by use of anorthogonal transformation.

Independent component analysis (ICA) decomposes a multivariate signalinto statistical independent non-Gaussian components. ICA could be usedfor feature selection and reduction. We stack our raw data vectors in amatrix where each row is an observation. ICA reduces the number ofcolumns or rearranges the information in the raw data into a smallernumber of features.

The system normalizes features using methods including but not limitedto z-score area under the curve (AUC).

For low level (internal) algorithms, a number of models may be employed:Deep learning techniques including conventional neural network (CNN) andtensor flow, Artificial Neural Networks; Support Vector Machines (SVM)including linear, non-linear; AdaBoost.

All of these machine learning methods are supervised binaryclassification models. A binary classifier is a numerical pipeline whichhas as input a numerical vector and outputs a binary decision, assigningthe input membership to one of two classes.

A deep learning model continually analyzes data with a logic structuresimilar to how a human would draw conclusions. To achieve this, deeplearning uses a layered structure of algorithms called an artificialneural network (ANN). The design of an ANN is inspired by the biologicalneural network of the human brain. This makes for machine intelligencethat's far more capable than that of standard machine learning models.

Differences between classical machine learning and AI versus deeplearning include: machine learning uses algorithms to parse data, learnfrom that data, and make informed decisions based on what has beenlearned; deep learning structures algorithms in layers to create an“artificial neural network” that can learn and make intelligentdecisions on its own; and deep learning is a subfield of machinelearning. While both fall under the broad category of artificialintelligence, deep learning powers the most human-like artificialintelligence.

The system may partition the data, with approximately 70% used fortraining and 30% for testing. The process may be repeated multiple timesto test if data is independent and identically distributed and isexchangeable. Such processing can help to evaluate intAI (internal AI)performance by calculating mean and confidence interval for intAIperformance. FIG. 9 shows one intAI architecture with feature extractionand selection (Extraction) prior to executing intAI.

The top level in the AI architecture of FIG. 9 is “fusion.” At thislevel, the system fuses internal (low-level) algorithms results(decisions) to obtain a final decision. With selected Classifiers, thesystem computes a weighted sum of Classifier decisions/score. The systemmakes a global decision by comparing this sum to a threshold.

From FIG. 9, there is provided a fluorescence path and a reflectancepath, where fluorescence passes fluorescence image readings to an inputdata conditioning element, which in turn passes data to featureextraction and selection modules. Reflectance passes reflectance imagereadings to signal conditioning modules to align data usinginterpolation grid and filter invalid data using data quality filterssuch as SNR (signal-to-noise ratio). If features level fusion offluorescence and reflectance data is desired, a similar data processingchain could be applied to reflectance data as well. The features comingfrom fluorescence and reflectance data are fused. After normalizationthe fused features are fed to a machine learning model. An alternaterepresentation of the processing is presented in FIG. 11.

With respect to AI training and internal Models Selection Process, thesystem may perform an exhaustive search and optimization to find bestinternal and fusion models. For the pipeline, the system starts with amachine learning model such as is outlined above. The system may thenoptimize the parameters of the model, seeking to maximize performance.With respect to an intAI model selection, the system may performselection of the best classification pipeline model using an exhaustivesearch process over the possible combinations of algorithms and controlparameters for each stage. At each point in the exhaustive searchevaluation, the system may apply a data partitioning to use a portion ofdata for optimization and the remaining for cross-validation test. Themain performance selection criterion is the averagesensitivity/specificity for all the cross-validation tests. The systemmay perform deeper analysis on Classifiers, which may be individualsoftware or hardware components or modules or combinations thereof, thathave passed a performance threshold (e.g., find an operating point onthe ROC with at least 95% sensitivity and maximum specificity amongother Classifiers).

The final model is fine-tuned using the training set for bestperformance. After data partitioning, a portion of data for AI trainingand the remaining for validation of trained AI models. The parameters ofeach stage may be fine-tuned around the values selected in the previousstep. The system may run this process multiple times to select the modelwith the best performance. The architecture of the models is the samebut the control parameters for each stage are data-driven and determinedby partitioning of the training data set. The system may validate thefinal model using the hold-out data set.

The design may include a wireless, portable high-speed device forassessment of food samples which is deployable to current fieldmeasurement demands. One such system may employ, for example three modesof spectral measurement:

-   -   Visible-NIR spectroscopy that surpasses human vision capability        (with spectral resolution at nm level) and facilitate        compositional analysis    -   Fluorescence spectroscopy, focusing on molecular structure as        well as protein, and harmful biomaterial detection and analysis        (e.g. toxins, spoilage, harmful bacteria)    -   IR spectroscopy to characterize chemometric aspects of our        specimens, such as water, protein, fat and carbohydrate        characteristics using classical absorption spectroscopy.

The present system employs multimode settings to cross-validate eachmode of measurement. For instance, the system can acquire purefluorescence measurements independent of light absorption (color) byreflectance and fluorescence in concurrent measurements. By analyzingfood samples, for example, using multimode methods, the system can moreaccurately differentiate the target of interest, and can analyzesubstantially more information, thus addressing a wider range ofcharacteristics and drawing deeper and more discrete conclusions (i.e.more targeted and valuable signatures). The AI algorithm may trainitself over time to be more efficient, where more efficient means higheraccuracy and faster assessment.

One example of multimode spectral measurement involves the measurementof pure fluorescence spectra independent of light absorption. Naturalfluorescence in food samples can be excited in multiple wavelengthranges. Examples include 278 nm (targeting Vitamin B2, tyrosine, andtryptophan), 305 nm (Targeting Vitamin B6, Vitamin E, and ATP), 365 nm(NADH, Vitamin A), 395 nm (hematoporphyrin), and 405 nm (chlorophyll).However, individual food sample may absorb light differently atexcitation or emission wavelengths. By independently characterizingreflectance spectra, the system can minimize the absorption contributionto the fluorescence spectra and purify fluorescence spectral signatures.

A second example of multimode operation (multi excitation fluorescence)is to more effectively unmix the fluorophores contributions of the foodsample. Usually, emission spectra of natural fluorophores are broad andoverlap to other natural fluorophores. Multiple excitation wavelengthshelps to differentiate individual fluorophores. Some of the moleculeshave specific absorption characteristics that can be individuallycalculated and to be used to improve fluorescence unmixing progress.Thus according to the present design, multimode may include using asingle technique in varying ways, such as at different frequencies orwavelengths.

Machine Learning operation of the present system can identify whichspectral features are important to differentiate between biologicalsamples such as food samples. Such machine learning helps the Classifierto weight specific molecular, compositional components relative to eachother for final classification. Machine learning also trains the expertsystem which combination of compositional, molecular, or chemicalcomponents becomes relevant and potentially important for classificationoptimization. Artificial intelligence in the system can establish astrategy where the classification can be optimized for either speedand/or accuracy by filtering most differentiating spectral features andremoving the redundant data.

A general overview of the present design is thus presented in FIG. 10.At point 1001, the system determines the biological sample(s) forevaluation. For example, the system may be desired to process only fish,or may process beef and pork, or may process fruits, vegetables, andoccasional meats of various kinds. This may be established by users orautomatically, such as by a user being offered options and selectingthose applicable. At point 1002, the system determines, using thecomponents offered and/or other means, attributes of samples forexamination and modes of examination. For example, a sample may call forexamination using Raman spectroscopy and infrared scanning, either byexperience or based on previous observations. In some instances, thesamples be examined may have no known best mode used, and thusexperimentation may be required to determine the desired use of mode Xon sample Y. Point 1003 is optional, wherein the system examines samplesto determine sample profiles and profile attributes of interest. Forexample, a sample with a particular known contaminant may be examinedusing speckle imaging and it may be determined that when examining at aparticular wavelength, the presence of the contaminant becomesparticularly pronounced, and thus all samples may be examined usingspeckle imaging at the particular wavelength to determine the presenceof the contaminant Alternately, if a biological sample such as a plantfrom a particular location exhibits an attribute under infrared imaging,similar plant samples may be examined using similar infrared imaging.

Point 1004 calls for identifying best modes and ensuring the best modesare available in the design. The various modes shown in FIG. 4 may beemployed, but other modes may be provided as suggested. In someinstances, examination in a single mode at various frequencies,wavelengths, or other measurement quantities may be employed. Such modesand examination attributes may be offered according to an examinationand analysis protocol. If it is determined that samples of interest mustbe examined at wavelength P in mode Q, mode Q must be offered and mustbe able to operate at wavelength P. Point 1005 represents generally theinitiation of production, i.e. the examination of multiple samplesaccording to the present design, wherein the device and modes arecalibrated. Point 1006 calls for examining samples using the device inthe desired modes using the desired attributes, or in other words,according to the examination and analysis protocol.

At point 1007, the system processes results, including makingassessments as to presence or absence of attributes, authenticationprobabilities, and so forth. Such processing employs the artificialintelligence and machine learning described herein. At point 1008, thesystem may evaluate and assess results, again using known attributes,machine learning, artificial intelligence, and/or other techniques.Results from this step importantly are fed back to point 1004,conceptually representing decisions to alter the examination andanalysis protocol as well as the mode or modes employed. As an example,the system may process thousands of samples of beef using a givenprotocol, such as examining using reflectance multiwavelength imaging atthree different wavelengths. However, examination at these wavelengthsmay offer limited results, such as a failure to determine the cut ofbeef being examined. In other words, results provided may beinconclusive. As a result, the system may augment the protocol andexamination by adding a different mode or may add a wavelength to thethree wavelengths used for examination. Thus the present system employsfeedback of determined results to improve the overall protocol and theoverall examination and analysis process, and the protocol establishedmay be dynamically changed depending on circumstances encountered.

FIG. 11 is a general overview of the processing modules employed. Moreor different modules may be employed. From FIG. 11, processor 1101controls all processing, including applicable machine learning,artificial intelligence, and the like. Point 1102 represents readingstaken, which are received by the processing arrangement 1103. Processingarrangement 1103 includes feature extraction module 1104 and classifiermodule 1105. The readings are received and distributed to the variousinternal AI training models 1106 a through 1106 m which generallyidentify known aspects and attributes from the readings based onexperience and/or prior readings. A single internal AI training modelmay be employed or offered. Fusion module 1107 fuses the results fromthe various internal AI training models 1106 a through 1106 m.Classifiers 1108 a through 1108 n classify the fused results asdescribed above, and overall results are provided at point 1109.Processor 1101 may then operate to provide the feedback shown in FIG.10, determining that different modes and/or different assessments may beemployed or different attributes examined, for example.

Thus according to one embodiment, there is provided a biological sampleinspection apparatus, comprising an illumination hardware arrangementcomprising transmission and sensing hardware, the illumination hardwarearrangement configured to inspect a biological sample using at least twomodes from a group comprising a fluorescence imaging mode, a reflectanceimaging mode, a scattering imaging mode, and a Raman imaging mode, andprocessing hardware configured to operate the illumination hardwarearrangement according to a protocol comprising inspection settings ofthe at least two modes, wherein the processing hardware receives scanresults from the illumination hardware arrangement and identifiesattributes of the biological sample. The processing hardware isconfigured to employ the attributes of at least one biological sample toalter the protocol.

According to a further embodiment of the present design, there isprovided a method for inspecting at least one biological sample,comprising determining a plurality of inspection modes for inspectingthe at least one biological sample using a multimode inspectionapparatus, determining an inspection protocol for inspecting the atleast one biological sample, wherein the inspection protocol comprisesinspection settings for the plurality of inspection modes, inspectingthe at least one biological sample using the multimode inspectionapparatus according to the protocol, and altering the protocol based oninspection results for multiple biological samples.

According to another embodiment of the present design, there is provideda biological sample inspection apparatus configured to inspect abiological sample for issues, comprising illumination hardwarecomprising transmission and sensing hardware configured to illuminateand sense attributes of the biological sample, the illumination hardwareconfigured to inspect the biological sample using multiple inspectionconfigurations from at least one of a fluorescence imaging mode, areflectance imaging mode, a scattering imaging mode, and a Raman imagingmode, and processing hardware configured to operate the illuminationhardware according to a protocol comprising inspection settings for themultiple inspection configurations, wherein the processing hardwarereceives scan results from the illumination hardware and identifiesattributes of the biological sample. The processing hardware isconfigured to employ the attributes of at least one biological sampleand alter the protocol based on the attributes of the one biologicalsample.

In one or more exemplary designs, the functions described may beimplemented in hardware, software, firmware, or any combination thereof.If implemented in software, the functions may be stored on ortransmitted over as one or more instructions or code on acomputer-readable medium. Computer-readable media includes both computerstorage media and communication media including any medium thatfacilitates transfer of a computer program from one place to another,i.e. may include transitory and/or non-transitory computer readablemedia. A storage media may be any available media that can be accessedby a computer. By way of example, and not limitation, suchcomputer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium that can be used to carry or store desiredprogram code in the form of instructions or data structures and that canbe accessed by a computer. Also, any connection is properly termed acomputer-readable medium. For example, if the software is transmittedfrom a website, server, or other remote source using a coaxial cable,fiber optic cable, twisted pair, digital subscriber line (DSL), orwireless technologies such as infrared, radio, and microwave, then thecoaxial cable, fiber optic cable, twisted pair, DSL, or wirelesstechnologies such as infrared, radio, and microwave are included in thedefinition of medium. Disk and disc, as used herein, includes compactdisc (CD), laser disc, optical disc, digital versatile disc (DVD),floppy disk and blu-ray disc where disks usually reproduce datamagnetically, while discs reproduce data optically with lasers.Combinations of the above should also be included within the scope ofcomputer-readable media.

The foregoing description of specific embodiments reveals the generalnature of the disclosure sufficiently that others can, by applyingcurrent knowledge, readily modify and/or adapt the system and method forvarious applications without departing from the general concept.Therefore, such adaptations and modifications are within the meaning andrange of equivalents of the disclosed embodiments. The phraseology orterminology employed herein is for the purpose of description and not oflimitation.

What is claimed is:
 1. A biological sample inspection apparatus,comprising: an illumination hardware arrangement comprising transmissionand sensing hardware, the illumination hardware arrangement configuredto inspect a biological sample using at least two modes from a groupcomprising: a fluorescence imaging mode; a reflectance imaging mode; ascattering imaging mode; and a Raman imaging mode; and processinghardware configured to operate the illumination hardware arrangementaccording to a protocol comprising inspection settings of the at leasttwo modes, wherein the processing hardware receives scan results fromthe illumination hardware arrangement and identifies attributes of thebiological sample; wherein the processing hardware is configured toemploy the attributes of at least one biological sample and alter theprotocol.
 2. The biological sample inspection apparatus of claim 1,wherein the biological sample comprises a food product.
 3. Thebiological sample inspection apparatus of claim 1, wherein theprocessing hardware comprises a processor, at least one trainedartificial intelligence module, and at least one classifier.
 4. Thebiological sample inspection apparatus of claim 1, wherein the protocolcalls for inhibiting scanning performance in one of the two modes underspecific established circumstances.
 5. The biological sample inspectionapparatus of claim 1, wherein the protocol is determined in part basedon an identification of particular attributes expected to be associatedwith the biological sample when examined using the at least two modes.6. The biological sample inspection apparatus of claim 1, wherein thebiological sample inspection apparatus determines authenticity of thebiological sample.
 7. A method for inspecting at least one biologicalsample, comprising: determining a plurality of inspection modes forinspecting the at least one biological sample using a multimodeinspection apparatus; determining an inspection protocol for inspectingthe at least one biological sample, wherein the inspection protocolcomprises inspection settings for the plurality of inspection modes;inspecting the at least one biological sample using the multimodeinspection apparatus according to the protocol; and altering theprotocol based on inspection results for multiple biological samples. 8.The method of claim 7, further comprising calibrating the multimodeinspection apparatus.
 9. The method of claim 7, wherein the at least onebiological sample comprises a food product.
 10. The method of claim 7,further comprising assessing the biological sample subsequent to theinspecting.
 11. The method of claim 10, further comprising: establishingtraining sets and classifier attributes used in the assessing; andimproving the training sets and classifier attributes based oninspection and assessment of multiple biological samples.
 12. The methodof claim 7, wherein the protocol is determined in part based on anidentification of particular attributes expected to be associated withthe biological sample when examined using the at least two modes. 13.The method of claim 7, wherein the method determines authenticity of theat least one biological sample.
 14. A biological sample inspectionapparatus configured to inspect a biological sample for issues,comprising: illumination hardware comprising transmission and sensinghardware configured to illuminate and sense attributes of the biologicalsample, the illumination hardware configured to inspect the biologicalsample using multiple inspection configurations from at least one of afluorescence imaging mode, a reflectance imaging mode, a scatteringimaging mode, and a Raman imaging mode; and processing hardwareconfigured to operate the illumination hardware according to a protocolcomprising inspection settings for the multiple inspectionconfigurations, wherein the processing hardware receives scan resultsfrom the illumination hardware and identifies attributes of thebiological sample; wherein the processing hardware is configured toemploy the attributes of at least one biological sample and alter theprotocol based on the attributes of the one biological sample.
 15. Thebiological sample inspection apparatus of claim 15, wherein thebiological sample comprises a food product.
 16. The biological sampleinspection apparatus of claim 15, wherein the processing hardwarecomprises a processor, at least one trained artificial intelligencemodule, and at least one classifier.
 17. The biological sampleinspection apparatus of claim 15, wherein the protocol calls forinhibiting scanning performance in at least one mode under specificestablished circumstances.
 18. The biological sample inspectionapparatus of claim 15, wherein the protocol is determined in part basedon an identification of particular attributes expected to be associatedwith the biological sample when examined using the at least one mode.19. The biological sample inspection apparatus of claim 15, wherein thebiological sample inspection apparatus determines authenticity of thebiological sample.